In order to use computer vision technology to accurately detect the fall of the elderly, aiming at the incompleteness of existing fall detection algorithms caused by artificial designing of features and the problems in the fall detection process such as the difficulty of separating foreground and background, the confusion of objects, the loss of moving objects, and the low accuracy of fall detection, a deep learning fall detection algorithm with the fusion of human motion information was proposed to detect the fall state of human body. Firstly, foreground and background were separated by the improved YOLOv3 network, and human object was marked by minimum bounding rectangle according to the detection results of YOLOv3 network. Then, by analyzing the motion features in the process of human fall, the motion features of human body were vectorized and transformed into the motion weight information between 0 and 1 through the Sigmoid activation function. Finally, in order to classify human falls, the motion features and the features extracted by Convolutional Neural Network (CNN) were spliced and fused through the fully connected layer. The proposed fall detection algorithm was compared with human object detection algorithms such as background difference, Gaussian mixture, VIBE (VIsual Background Extractor), Histogram of Oriented Gradient (HOG) and human fall judgment schemes such as threshold method, grading method, Support Vector Machine (SVM) classification, CNN classification, and tested under different lighting conditions and the interference of mixed daily noise motion. The results show that the proposed algorithm is superior to traditional human fall detection algortihms in environmental adaptability and fall detection accuracy. The proposed algorithm can effectively detect the human body in the video and accurately detect the fall state of human body, which further verifies the feasibility and efficiency of the deep learning recognition method with the fusion of motion information in the video fall behavior analysis.